Cyb/metadata table by alpass163gmail · Pull Request #18127 · apache/iotdb

alpass163gmail · 2026-07-06T10:37:57Z

This PR introduces a lease-based self-fencing framework to enable high
availability for table metadata operations. It prevents DataNodes from
serving stale schema during network partitions.

Problem

Currently, in a cluster deployment (e.g., 1 ConfigNode, 3 DataNodes),
metadata operations lack true high availability. If a single DataNode (DN)
crashes or experiences a network partition, execution of DDL procedures
(such as Create/Alter/Drop Table, View management, TTL adjustments, and
Drop Database) will directly fail and trigger a rollback.

Solution

Lease Mechanism

MetadataLeaseManager (DataNode): Tracks lease via ConfigNode heartbeats.
Uses monotonic clock. Self-fences (clears cache, blocks reads/writes) if no
heartbeat received within metadata_lease_fence_ms (T_fence).
MetadataLeaseFencedException: Thrown when operations are blocked on a
fenced DataNode.

Broadcast Coordination

DataNodeContactTracker (ConfigNode): Records time of last successful
heartbeat response per DataNode. Separately maintained from load-balancing
samples to ensure correctness.
MetadataBroadcastVerdict: Pure decision logic — PROCEED if all unacked
DataNodes have been silent for T_proceed = T_fence + margin, WAIT
otherwise, FAIL when retry budget exhausted.
ClusterCachePropagator: Broadcasts cache invalidations with retry loop,
waiting up to T_proceed for unresponsive DataNodes to prove self-fenced.

Schema Change Integration

Procedures now propagate metadata invalidations via
ClusterCachePropagator before proceeding.
Pre-deletion marker (PreDeleteTsTable) added for safe table state
transitions.
Rollback mechanism (RollbackPreDeleteTablePlan) for failed schema
changes.

Configuration

New config: metadata_lease_fence_ms (default in
iotdb-system.properties.template).

Testing

Unit tests for MetadataLeaseManager, DataNodeContactTracker,
MetadataBroadcastVerdict, ClusterCachePropagator
Lease integration tests for DataNodeTableCache, PartitionCache,
ClusterAuthorityFetcher
New HA IT: IoTDBTableDDLHAIT

Key Components Added

MetadataLeaseManager (DataNode): Tracks the lease and performs
self-fencing upon expiration.
DataNodeContactTracker (ConfigNode): Tracks the last successful heartbeat
timestamp for each DataNode.
ClusterCachePropagator (ConfigNode): Broadcasts cache invalidations with
a fencing-aware retry mechanism.
MetadataBroadcastVerdict (ConfigNode): Decides when unacknowledged
DataNodes are logically safe to skip.
MetadataLeaseFencedException (node-commons): The explicit exception
thrown on a fenced DataNode.
PreDeleteTsTable (node-commons): Represents the marker table state to
ensure safe schema deletion.

…lity of table metadata operation procedure

… lease frame

…ager, add the scheduled task to check the status of metadata

# Conflicts: # iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/env/ConfigNodeProcedureEnv.java # iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/DataNodePipeMessages.java # iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/DataNodeSchemaMessages.java

…m CommonConfig to the IoTDBConfig

Caideyipi · 2026-07-07T07:54:51Z

I reviewed this PR against the HA requirement/design docs. The direction looks good, but I found several issues that should be addressed before merge:

The DataNode fencing decision is not checked synchronously on access.

MetadataLeaseManager.isFenced() only checks whether metadataState != NORMAL. Lease expiration is moved from NORMAL to NEED_CLEAR only by the scheduled checkLeaseStatus() task. If check_dn_lease_status_interval_ms is configured larger than the ConfigNode-side T_fence + margin, the ConfigNode may proceed with a metadata change while the partitioned DataNode is still in NORMAL and can keep serving stale cache. Please make isFenced() / failIfMetadataLeaseFenced() check the elapsed heartbeat age and CAS the state to NEED_CLEAR immediately when the lease is expired.

A reachable but already-fenced DataNode can make the HA broadcast fail.

DataNodeInternalRPCServiceImpl.updateTable() calls DataNodeTableCache.preUpdateTable(), which throws MetadataLeaseFencedException while the DataNode is fenced. On the ConfigNode side, ClusterCachePropagator treats any response other than SUCCESS_STATUS and CAN_NOT_CONNECT_DATANODE as FAIL. This means a DataNode that has correctly self-fenced but is still reachable, or is recovering in NEED_CLEAR / PULLING, can fail the DDL instead of being considered safe. Please return/handle METADATA_LEASE_FENCED explicitly and let the propagator treat it as a safe fenced response.

The advertised supportsFencing compatibility guard is missing.

The design says the ConfigNode should track whether each DataNode supports self-fencing, with default false for old nodes. Current ClusterCachePropagator only uses heartbeat age. During rolling upgrade or mixed-version operation, an old DataNode that does not self-fence can still be considered safe after T_proceed, which is unsafe. Please add capability reporting/tracking and only skip unreachable DataNodes that are known to support fencing.

The fetchTables Thrift change is not backward compatible.

fetchTables(map<string, set<string>> fetchTableMap) was changed to fetchTables(map<string, set<string>> fetchTableMap, byte tableNodeStatus). This breaks old DataNode/new ConfigNode compatibility: an old caller will not set the new argument, and the server can parse it as the default byte value 0 (PRE_CREATE), which then goes into the unsupported branch in ConfigManager.fetchTables(). Please use a new RPC or an optional/defaulted argument that preserves the old behavior as USING.

There are new raw English log/message literals in main code.

For example, ConfigNodeRPCServiceProcessor.reloadCacheAfterLeaseRecovery() logs Execute getMetaDataCache with result {} directly. This repo requires user/operator visible strings to go through the i18n message constants. Please move the new literals to the proper message classes.

Overall, the main concept is sound, but the above issues affect the safety and compatibility of the lease-based HA mechanism.

JackieTien97

ProcedureMessages.INVALIDATING_CACHE_FOR_TABLE_WHEN_DROPPING_TABLE is dead code, shoule be deleted
ConfigNodeRPCServiceProcessor.java:344 —— "Execute getMetaDataCache with result {}" shoule use i18n to refactor

# Conflicts: # iotdb-core/datanode/src/test/java/org/apache/iotdb/db/schemaengine/table/DataNodeTableCacheTest.java

Yaobin Chen added 6 commits July 3, 2026 09:36

introduce the lease and self-fence mechanism to support high availabi…

d8d2591

…lity of table metadata operation procedure

set the MetadataLeaseFenceMs in UT envSetUp, remove the effect of the…

1ad5281

… lease frame

fix the problems, main modification is the change of MetadataLeaseMan…

5b26f9b

…ager, add the scheduled task to check the status of metadata

fix i18n

8b604cf

fix: remove the configuration 'check_dn_lease_status_interval_ms' fro…

2dd3759

…m CommonConfig to the IoTDBConfig

JackieTien97 requested changes Jul 7, 2026

View reviewed changes

Yaobin Chen added 2 commits July 7, 2026 19:06

add the IT, include the one DN with readOnly status in HA

7b45cb9

fix i18n, introduce the enum TsTableMarker

f684bde

JackieTien97 reviewed Jul 8, 2026

View reviewed changes

fix i18n

9032361

JackieTien97 approved these changes Jul 8, 2026

View reviewed changes

Yaobin Chen added 5 commits July 8, 2026 17:23

Merge branch 'refs/heads/master' into cyb/metadata-table

74deae7

# Conflicts: # iotdb-core/datanode/src/test/java/org/apache/iotdb/db/schemaengine/table/DataNodeTableCacheTest.java

spotless the file

8ddf01b

replace the sendAsyncRequestWithRetry

333a5eb

filter the unknown DNs with fenced status before CN sendRpc

ac84254

fix the BROADCAST_RPC_TIMEOUT_MS and retry times

c0eef3d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cyb/metadata table#18127

Cyb/metadata table#18127
alpass163gmail wants to merge 14 commits into
apache:masterfrom
alpass163gmail:cyb/metadata-table

alpass163gmail commented Jul 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Caideyipi commented Jul 7, 2026

Uh oh!

JackieTien97 left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alpass163gmail commented Jul 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Caideyipi commented Jul 7, 2026

Uh oh!

JackieTien97 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JackieTien97 left a comment •

edited

Loading